privacy-filter is an inline HF Transformers token-classification server
(`pipeline(..., device_map="auto")`) with no memory bound. Under steady
traffic the CUDA caching allocator's reserved memory ratchets up and is
never released, so the process slowly hoards the GPU it shares with
Qwen3-VL, FLUX, embeddings, reranker and whisper (GPU 7). Observed ~93 GB
held on an H200 for a model that needs ~1-2 GB.

As privacy-filter fills the card (free ~50 GB -> ~0 over 1-2 days) the
largest co-tenant, Qwen3-VL (~49 GB at --gpu-memory-utilization 0.35),
can no longer load and crash-loops with
`torch.AcceleratorError: CUDA error: out of memory`. The same leak OOM'd
embeddings/whisper on 2026-05-25. Hits both small-models hosts (gpu11,
gpu02) since they run identical config.

Fix (inline server + container env):
- empty_cache() after every request (core fix): returns cached-but-unused
  CUDA blocks to the driver so reserved memory stops ratcheting.
- set_per_process_memory_fraction(GPU_MEM_FRACTION, 0) (fail-safe): hard
  ceiling so the process self-OOMs/restarts instead of starving neighbours.
  Default 0.10 (~14 GB on a 140 GB H200), env-tunable.
- torch.inference_mode() around inference: no autograd state retained.

Interim mitigation already applied by recreating the container, which
frees the leaked VRAM but recurs in ~1-2 days; this makes it permanent.
Ship via the normal tag + compose/up redeploy of small-models.yaml.

…_segments)

Addresses the code review of the first cut:

- Root cause now fixed at the source: PYTORCH_CUDA_ALLOC_CONF=expandable_segments
  lets the CUDA allocator shrink reserved segments instead of ratcheting up.
- Drop per-request torch.cuda.empty_cache(): a synchronizing cudaFree on the hot
  path stalled the shared GPU and the co-located models it was meant to protect.
  A 30s watchdog thread now releases idle blocks off the request path.
- Real fail-safe instead of a silent 500-storm: the watchdog hard-restarts the
  container (os._exit -> restart:unless-stopped) if this process's reserved VRAM
  exceeds GPU_MEM_LIMIT_GB, and an acute CUDA-OOM in a request also exits. The
  prior "self-OOMs and restarts" comment was false — a caught OOM returned 500
  while the process stayed up behind a still-healthy /v1/models probe.
- Drop set_per_process_memory_fraction: the 0.10 (~14GB) guess could OOM legit
  batch_size=32 requests, and device_map="auto" planned against the full card
  and ignored the cap anyway. Bound the work via PRIVACY_BATCH_SIZE instead;
  inputs are NOT truncated (a privacy filter must see the whole text).
- device=0 instead of device_map="auto" (no accelerate planner mismatch).
- Drop torch.inference_mode(): redundant with the pipeline's internal no_grad
  and stricter (risked raising under trust_remote_code custom models).
- Tolerant env parsing + clamps so a malformed knob can't crash-loop boot.

Validated: small-models.yaml parses and the embedded server.py compiles.